1. Field of the Invention
The present invention relates to electronic systems for generating special effects in color television and color motion pictures and, more particularly, to systems for replacing selected portions of a foreground image with a background image when desired.
2. History of the Prior Art
There are in existence various systems for keying out selected portions of a foreground image such as may be provided by a foreground television camera and replacing the keyed out portions with a background image such as may be provided by a background television camera. Examples of such systems are provided by U.S. Pat. No. 3,778,542, Hanseman, Blue Screen Traveling Matte System, by U.S. Pat. No. 3,595,987, Vlahos, Electronic Composite Photography, by U.S. Pat. No. 3,560,638, Skrydstrup, Chroma Keying System Utilizing Remote Controlled Chroma Keyer, and by a copending application, Ser. No. 443,388, now abandoned filed Feb. 19, 1974, Luminance Key Amplifier, Mendrala.
The Hanseman patent provides an example of a traveling matte system in which the foreground and background signals are algebraically combined such as by use of suppression networks to produce the composite signal. In the particular arrangement shown in the Hanseman patent a color difference signal is derived by subtracting the sum of the red and green components of the foreground signal from twice the value of the blue component. The foreground signal is selectively suppressed by a network which acts to subtract the blue component from the total signal. The background signal components are applied to multipliers where they are selectively suppressed by the color difference signal.
The Vlahos patent provides another example of a traveling matte system. In the arrangement disclosed in the Vlahos patent a color difference signal is derived by subtracting either the green or red component of the foreground signal from the blue component. The foreground and background signals are then gated to an extent proportional to the relationship of the blue and green (or red) components of the foreground signal to provide the composite signal. Unlike Hanseman, the Vlahos arrangement has no provision for varying the keying color in the foreground image. A particular shade of blue is used in all cases.
The Skrydstrup patent provides an example of a keying system in which the output is effectively switched between the foreground and background signals to generate the composite signal. The switching is done electronically and in response to comparison of a selected color in the foreground signal to a reference level. The selected color in the foreground signal is identified by an arrangement which matrixes the red, green and blue components of the foreground signal to provide a luminance signal which is then subtracted from the red and blue components by differential amplifiers. The resulting differences are altered in voltage controlled amplifiers by the outputs of a sine/cosine potentiometer, than combined to provide a signal identifying the presence of the selected color in the foreground signal.
Copending application Ser. No. 443,388 provides another example of a keying system in which the output is effectively switched between the foreground and background signals to generate the composite picture. In that arrangement the red, green and blue components of the foreground signal are matrixed to provide an average thereof which is then compared with a reference value. The results of the comparison are used to key either the foreground signal or the background signal to the output.
Presently known arrangements for mixing foreground and background signals to provide a composite signal suffer from a number of limitations which may prove to be distinctly disadvantageous in certain applications of such systems. One problem with such systems is that they cannot provide for both a shadow capability and the elimination of the outline which frequently occurs between the remaining parts of the foreground image and those parts replaced by the background image. Thus while the arrangement disclosed in copending application Ser. No. 443,388 is effective in preventing the outline, among other reasons because of its processing of the video signals prior to the encoding thereof, such arrangement is a keying system and is therefore incapable of preserving shadow information in the foreground image. Since keying systems by their very nature couple either the foreground signal or the background signal to the output, shadows which appear in portions of the foreground image replaced by the background image disappear since there is no way of varying the background image to preserve the shadows. The background image is either present or not depending on the electronic switching of such systems.
Traveling matte systems, on the other hand, are capable of preserving shadow information because of their capability of proportionally mixing the foreground and background images in the generation of the composite image. Thus where a shadow appears in a portion of the foreground image to be replaced by the background image, various schemes can be used to substitute the background image in such a way that the shadow from the foreground remains. This feature is particularly important, for example, in scenes where the foreground image comprises one or more persons who are photographed against a uniform, colored background and who are to be superimposed on a background taken from a miniature set. If the scene takes place in bright sunlight, for example, the persons in the foreground will almost always cast noticeable shadows. The ability to preserve such shadows when the background image is substituted for the blank background of the foreground image is very important if realism is to be achieved.
While traveling matte systems such as of the type shown in the Hanseman and Vlahos patents are capable of preserving some shadow information from the foreground image, such systems also have a disturbing matte outline between the foreground and background images. The matte outline typically appears as a black or white margin which divides the foreground and background images. In the case of the persons from the foreground image being superimposed on a background image such as may be made from a miniature set, the matte outline appears as a black or white outline surrounding and separating the persons from the background in the composite picture. The matte outline may sometimes be reduced or even eliminated in traveling matte systems by increasing the gain of the background signal and in some cases the foreground signal prior to the combination of such signals to form the composite picture. However as the gain of the background signal is increased the ability to discriminate between shadowed and non-shadowed portions of the foreground image is diminished to the point where the shadow capability of the system is completely lost as a result of substantially minimizing or eliminating the matte outline.
Moreover, even where the gain is set relatively low in traveling matte systems so as to preserve the shadows at the expense of the presence of a matte outline. the shadow capability of such systems is limited. Thus, vivid shadows appearing in the foreground image are typically much less noticeable when reproduced in the background. This is due in part to the non-linear transfer characteristic introduced by most television cameras. While the transfer characteristic of any natural scene is such that the light output on reflection varies linearly with the intensity of input light on the scene, the same is not necessarily true at the television receiver. Picture tubes typically impart a nonlinear transfer characteristic to the darker shades, particularly black. Most television cameras compensate for this by use of a gamma amplifier. However while certain advantages are achieved, the transfer characteristic at the camera output is non-linear. As a result shadowed portions of the original image which are different enough from illuminated portions of the image so as to readily appear in the composite image become less distinguishable from the lighter portions of the image, requiring relatively high levels of illumination of the foreground image before the shadows will be transferred to the composite image. Compounding the problem is the fact that shadowed portions occur in a high gain region of the resulting transfer characteristic, resulting in the presence of considerable noise in the shadows.
Further problems lie in the manner in which the color difference signal is generated in prior art systems. As previously noted the arrangement shown in the Vlahos patent keys only on a particular shade of blue, and the keying color is not variable. In the system shown in the Hanseman patent the keying color is adjustable via a potentiometer. However, such adjustments cannot be made in locations remote from the system because of the presence of video signals in the adjustment process. In the arrangement shown in the Skrydstrup patent the keying color is remotely adjustable via a sine/cosine potentiometer, the outputs of which are applied to voltage controlled amplifiers. However, the resulting color difference signal is subject to drift and other inaccuracies, among other reasons because of the particular circuitry used in Skrydstrup. The differences between the luminance signal and the red and blue color components are obtained using differential amplifiers, following which the resulting difference signals are amplified in the voltage controlled amplifiers in accordance with the signals from the sine/cosine potentiometer. The voltage controlled amplifiers are difficult to control and are subject to drift. Moreover, the gain provided by such amplifiers varies in non-linear fashion.
Accordingly, it would be desirable to provide a system in which the color difference signal is generally immune from drift and other variations and inaccuracies.
It would furthermore be desirable to provide a system which eliminates the matte outline while at the same time preserving shadows in those portions of the foreground image replaced by the background image.
It would still furthermore be advantageous to provide a system in which relatively light shadows in the foreground image are readily detected and reproduced in the composite image without the necessity for greatly increasing the illumination in selected areas of the foreground image.
It would still furthermore be advantageous to provide a system in which shadows in the foreground image which are reproduced in the composite image can be controlled and adjusted electronically.